多视野特征表示的灰度图像彩色化方法

doi:10.16451/j.cnki.issn1003-6059.202207006

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (3359 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要图像彩色化是指预测灰度图像的颜色信息,虽然使用深度学习方法可自动地对灰度图像彩色化,但对图像中不同尺度目标的彩色化质量不高,尤其是在对复杂物体和小目标物体彩色化时,存在颜色溢出、误着色和图像颜色不一致的问题.针对上述问题,文中提出多视野特征表示的灰度图像彩色化方法.首先,设计多视野特征表示模块(Multi-field Feature Represented Block, MFRB),与改进的U-Net结合得到多视野特征表示U-Net.然后,将灰度图像输入U-Net中,并通过与判别器的对抗训练得到彩色图像.最后,利用VGG-19网络在不同尺度上计算图像的感知损失,提高图像彩色化结果的整体一致性.在不同类别的6个数据集上的实验表明,文中方法能有效提高彩色化图像质量,产生颜色更丰富、色调更一致的彩色图像,并在客观评价指标和主观感受上都较优.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	李洪安
	郑峭雪
	马天
	张婧
	李占利
	康宝生

关键词 ：图像彩色化, 生成对抗网络, 多视野特征表示, 感知损失

Abstract：Image colorization improves image quality by predicting color information of gray-scale images. Although the grayscale images can be colored automatically by deep learning methods, the colorization quality of targets with different scales in the images is not satifactory. Especially, the existing colorizing methods is confronted with problems of color overflow, mis-coloring and inconsistent image colors, while dealing with complex objects and small target objects. To address these problems, a method for image colorization of multi-field features representation is proposed in the paper. Firstly, the multi-field feature representation block(MFRB) is designed and combined with the upgraded U-Net to acquire multi-field feature representation U-Net. Then, a grayscale image is input into the U-Net and the color image is obtained by adversarial training with PatchGAN. Finally, the VGG-19 network is employed to compute the perceptual loss of pictures at different scales to enhance the general consistency of the image colorization results. Experimental results on six distinct datasets demonstrate that the proposed method successfully enhances the quality of colorized images and creates color images with richer colors and more consistent tones. The results of the proposed method outperform the main colorization algorithms in both quantitative assessment and subjective perception.

Key words： Image Colorization Generative Adversarial Networks Multi-field Feature Representation Perceptual Loss

收稿日期: 2022-05-17

ZTFLH:

TP 391.41

基金资助:国家自然科学基金项目(No.61902311)、陕西省自然科学基础研究计划项目(No.2022JM-508,2022JM-317)资助

通讯作者: 李洪安,博士,副教授,主要研究方向为计算机图形学、可视媒体计算.E-mail:honganli@xust.edu.cn.

作者简介: 郑峭雪,硕士研究生,主要研究方向为可视媒体计算、人工智能.E-mail:20208088025@stu.xust.edu.cn.
马天,博士,副教授,主要研究方向为三维仿真、图像处理、数据可视化.E-mail:matian@xust.edu.cn.
张婧,博士,讲师,主要研究方向为图形图像处理、智能信息处理.E-mail:zhangjing21@xust.edu.cn.
李占利,博士,教授,主要研究方向为智能信息处理、视觉计算与可视化.E-mail:lizl@xust.edu.cn.
康宝生,博士,教授,主要研究方向为计算机图形学、图像处理.E-mail:bskang@nwu.edu.cn.

引用本文:

李洪安, 郑峭雪, 马天, 张婧, 李占利, 康宝生. 多视野特征表示的灰度图像彩色化方法[J]. 模式识别与人工智能, 2022, 35(7): 637-648. LI Hong'an, ZHENG Qiaoxue, MA Tian, ZHANG Jing, LI Zhanli, KANG Baosheng. Multi-field Features Representation Based Colorization of Grayscale Images. Pattern Recognition and Artificial Intelligence, 2022, 35(7): 637-648.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202207006 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2022/V35/I7/637

[1] KHAN M U G, GOTOH Y, NIDA N. Medical Image Colorization for Better Visualization and Segmentation // Proc of the Annual Conference on Medical Image Understanding and Analysis. Berlin, Germany: Springer, 2017: 571-580.
[2] LIZUKA S, SIMO-SERRA E, ISHIKAWA H. Let There Be Color! Joint End-to-End Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transactions on Graphics, 2016, 35(4). DOI: 10.1145/2897824.2925974.
[3] ZHANG R, ISOLA P, EFROS A A. Colorful Image Colorization // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 649-666.
[4] DESHPANDE A, LU J J, YEH M C, et al. Learning Diverse Image Colorization // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2877-2885.
[5] HE M M, CHEN D D, LIAO J, et al. Deep Exemplar-Based Colorization. ACM Transactions on Graphics, 2018, 37(4). DOI: 10.1145/3197517.3201365.
[6] LEVIN A, LISCHINSKI D, WEISS Y. Colorization Using Optimization. ACM Transactions on Graphics, 2004, 23(3): 689-694.
[7] LIU Y F, XU K, YAN L Q. Adaptive BRDF-Oriented Multiple Importance Sampling of Many Lights. Computer Graphics Forum, 2019, 38(4): 123-133.
[8] HEU J H, HYUN D Y, KIM C S, et al. Image and Video Colorization Based on Prioritized Source Propagation // Proc of the 16th IEEE International Conference on Image Processing. Washington, USA: IEEE, 2009: 465-468.
[9] WANG Y L, LIU Y F, XU K. An Improved Geometric Approach for Palette-Based Image Decomposition and Recoloring. Computer Graphics Forum, 2019, 38(7): 11-22.
[10] 曹丽琴,商永星,刘婷婷,等.局部自适应的灰度图像彩色化.中国图象图形学报, 2019, 24(8): 1249-1257.
(CAO L Q, SHANG Y X, LIU T T, et al. Novel Image Colorization of a Local Adaptive Weighted Average Filter. Journal of Image and Graphics, 2019, 24(8): 1249-1257.)
[11] 李洪安,张敏,杜卓明,等.一种基于分块特征的交互式图像色彩编辑方法.红外与激光工程, 2019, 48(12): 293-298.
(LI H A, ZHANG M, DU Z M, et al. Interactive Image Color Editing Method Based on Block Feature. Infrared and Laser Engineering, 2019, 48(12): 293-298.)
[12] LI F, NG M K. Image Colorization by Using Graph Bi-laplacian. Advances in Computational Mathematics, 2019, 45(3): 1521-1549.
[13] ZHANG Q, XIAO C X, SUN H Q, et al. Palette-Based Image Recoloring Using Color Decomposition Optimization. IEEE Transactions on Image Processing, 2017, 26(4): 1952-1964.
[14] LI B, LAI Y K, JOHN M, et al. Automatic Example-Based Image Colorization Using Location-Aware Cross-Scale Matching. IEEE Tran-sactions on Image Processing, 2019, 28(9): 4606-4619.
[15] HU S M, LIANG D, YANG G Y, et al. Jittor: A Novel Deep Learning Framework with Meta-Operators and Unified Graph Execution. Science China Information Sciences, 2020, 63(12). DOI: 10.1007/s11432-020-3097-4.
[16] ZHANG S H, LI R L, DONG X, et al. Pose2Seg: Detection Free Human Instance Segmentation // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 889-898.
[17] LIANG Y, WANG X T, ZHANG S H, et al. PhotoRecomposer: Interactive Photo Recomposition by Cropping. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(10): 2728-2742.
[18] LI H A, ZHANG M, YU Z H, et al. An Improved pix2pix Model Based on Gabor Filter for Robust Color Image Rendering. Mathematical Biosciences and Engineering, 2022, 19(1): 86-101.
[19] 李洪安,郑峭雪,张婧,等.结合Pix2Pix生成对抗网络的灰度图像着色方法.计算机辅助设计与图形学学报, 2021, 33(6): 929-938.
(LI H A, ZHENG Q X, ZHANG J, et al. Pix2Pix-Based Grays-cale Image Coloring Method. Journal of Computer-Aided Design and Computer Graphics, 2021, 33(6): 929-938.)
[20] SANGKLOY P, LU J W, FANG C, et al. Scribbler: Controlling Deep Image Synthesis with Sketch and Color // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 6836-6845.
[21] YOO S, BAHNG H, CHUNG S, et al. Coloring with Limited Data: Few-Shot Colorization via Memory Augmented Networks // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2019: 11275-11284.
[22] JOHARI M M, BEHROOZI H. Context-Aware Colorization of Gray-Scale Images Utilizing a Cycle-Consistent Generative Adversarial Network Architecture. Neurocomputing, 2020, 407: 94-104.
[23] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature Pyramid Networks for Object Detection // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 936-944.
[24] ZHAO H S, SHI J P, QI X J, et al. Pyramid Scene Parsing Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 6230-6239.
[25] CHEN C F, FAN Q F, MALLINAR N, et al. Big-Little Net: An Efficient Multi-scale Feature Representation for Visual and Speech Recognition[C/OL].[2022-04-24]. https://arxiv.org/pdf/1807.03848.pdf.
[26] MENG Y, LIN C C, PANDA R, et al. AR-Net: Adaptive Frame Resolution for Efficient Action Recognition // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 86-104.
[27] JOHNSON J, ALAHI A, LI F F. Perceptual Losses for Real-Time Style Transfer and Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 694-711.
[28] RAD M S, BOZORGTABAR B, MARTI U V, et al. SROBB: Targeted Perceptual Loss for Single Image Super-Resolution // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2019: 2710-2719.
[29] CHEN L F, YANG Z, MA J J, et al. Driving Scene Perception Network: Real-Time Joint Detection, Depth Estimation and Semantic Segmentation // Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2018: 1283-1291.
[30] XIAO X, LIAN S, LUO Z M, et al. Weighted Res-UNet for High-Quality Retina Vessel Segmentation // Proc of the 9th International Conference on Information Technology in Medicine and Education. Washington, USA: IEEE, 2018: 327-331.
[31] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative Adversarial Nets // Proc of the 27th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2014: 2672-2680.
[32] NAZERI K, NG E, EBRAHIMI M. Image Colorization Using Generative Adversarial Networks // Proc of the International Conference on Articulated Motion and Deformable Objects. Berlin, Germany: Springer, 2018: 85-94.
[33] VITORIA P, RAAD L, BALLESTER C. ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution // Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2020: 2434-2443.
[34] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-Image Translation with Conditional Adversarial Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5967-5976.
[35] GUAN S, KHAN A A, SIKDAR S, et al. Fully Dense UNet for 2D Sparse Photoacoustic Tomography Artifact Removal. IEEE Journal of Biomedical and Health Informatics, 2019, 24(2): 568-576.
[36] IBTEHAZ N, RAHMAN M S. MultiResUNet: Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation. Neural Networks, 2020, 121: 74-87.
[37] GATYS L A, ECKER A S, BETHGE M.A Neural Algorithm of Artistic Style[C/OL]. [2022-04-24].https://arxiv.org/pdf/1508.06576.pdf.
[38] GATYS L A, ECKER A S, BETHGE M. Image Style Transfer Using Convolutional Neural Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2414-2423.
[39] ANWAR S, TAHIR M, LI C Y, et al. Image Colorization: A Survey and Dataset[C/OL].[2022-04-24]. https://arxiv.org/pdf/2008.10774.pdf.
[40] ZHANG R, ISOLA P, EFROS A A, et al. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 586-595.